Integration of control data into digital broadcast content for access to ancillary information

ABSTRACT

Ancillary information associated with one or more images embedded in a digital broadcast data stream may be accessed by a digital broadcast receiving device. The device may include a receiver unit, a processor, and a memory. The processor may execute program instructions to access the ancillary information. Control data may be associated with one or more visual components of one or more video images. The control data and images may be combined into a digital broadcast data stream. The device may receive the images and control data. The images may be presented on a display device. The receiving device may use the control data to present or store a perceptible indication that ancillary information is associated with the visual components in conjunction with presenting the images. The device may use the control data to present or store the ancillary information in response to an input signal.

CLAIM OF PRIORITY

This application is a continuation of U.S. patent application Ser. No.13/560,213, filed on Jul. 27, 2012, and entitled, “Integration ofControl Data into Digital Broadcast Content for Access to AncillaryInformation, which is a continuation of U.S. patent application Ser. No.12/128,096, filed May 28, 2008, (since issued as U.S. Pat. No. 8,239,896on Aug. 7, 2012) the entire contents of which are incorporated herein byreference.

FIELD OF THE INVENTION

Embodiments of this invention are related to digital broadcasting andmore specifically to updating data for services usable in conjunctionwith a digital broadcast signal.

BACKGROUND OF THE INVENTION

Since broadcast television was introduced in the 1930s, televisionstations and networks have been paid to advertise products and servicesin conjunction with scheduled programming. Such advertising is often inthe form of commercials that are broadcast during breaks in programcontent. A common problem with such commercials from the point of viewof the viewer is that they interrupt the programming that the viewerwould like to watch. Another problem with commercials from the point ofview of the advertiser is that it is difficult to track theeffectiveness of a given ad in encouraging a given viewer to purchasesomething advertised in a commercial.

A further problem has been the development of the digital videorecorders (DVR), which allows a viewer to “fast-forward” throughcommercials and thereby avoid their impact entirely. Many DVR systemsalso allow a viewer to “pause” or “rewind” a television program. Thisfeature is commonly used by DVR users when television viewing isinterrupted, e.g., by a telephone call. Some viewers may use a DVR topause or even rewind a program to get a closer look at something shownin the course of a broadcast.

To enhance revenue streams for broadcasters and to make advertising lessintrusive to viewers, some programs include advertising in a formreferred to herein as “product placement.” In this type of advertising,a product, or the logo or trademark for the product, appears in atelevision program as part of the program. For example, a program mayshow a character ostentatiously drinking from a can marked with the logoof a popular brand or soft drink or driving a particular make and modelof automobile, with the manufacturer's trademark prominently displayed.Although these advertisements are less intrusive than conventionaltelevision commercials, their impact on consumers may be lessened forthis very reason.

A major change in the nature of television for both advertisers andconsumers arose with the advent of digital television. Introduced in thelate 1990s, digital television (DTV) technology appealed to thetelevision broadcasting business and consumer electronics industries asoffering new consumer services and business opportunities that wereimpractical with previous analog television. Digital television is moreflexible and efficient than analog television. When properly used bybroadcasters, digital television allows higher-quality images and soundand more programming choices than analog does. In DTV moving images andsound are sent and received by means of discrete (digital) signals, incontrast to the analog signals used by analog television. Digitaltelevision includes, but is not limited to Digital TerrestrialTelevision (DTTV or DTT), which is an implementation of digitaltechnology to provide a greater number of channels and/or better qualityof picture and sound using aerial broadcasts to a conventional antenna(or aerial) as opposed to a satellite dish or cable connection.

The development of digital television has lead to many changes intelevision broadcasting and related industries. Many countries havemandated a change from an analog television signal format to a newdigital format. One example of such a digital television broadcaststandard was developed by the Advanced Television Systems Committee.With a conventional analog television broadcast, a video signalmodulates a carrier wave signal that is broadcast by a transmissiontower. A television set contains a receiver that detects broadcastsignals. The receiver includes a tuner that selects a particular channelaccording to its carrier frequency and a demodulator that extracts thevideo signal from the modulated carrier signal. With a digitaltelevision signal, the video signal is generated in a digital format oran analog video signal is converted to a digital format to produce adigital signal. The carrier wave is modulated according to the digitalsignal format, e.g., using vestigial sideband (VSB) modulation. This newformat allows data for additional digital services to be broadcast alongwith a regular television signal. Examples of such services may includereal time stock quotes, sports, weather and traffic updates and otherservices traditionally associated with delivery via two-way media, suchas the internet, digital wireless services or cellular telephoneservices.

Digital broadcasting allows broadcasters to provide ancillary data orcontrol data associated with program content. For example, some digitalradio broadcast signals include data identifying program content, suchas a program title or the name and/or artist of a song played over adigital broadcast. A digital radio receiver can display thisinformation, e.g., on a text display. Some systems can even use thisdata to allow a user to download content. This type of two-way activityis relatively easy to implement in digital radio, which is essentially aone dimensional audio medium. However, the multi-dimensional audio-videonature of television makes such two-way activity problematic for digitaltelevision systems.

The delivery of digital services by a primarily one-way medium throughdigital broadcast presents certain challenges and problems. Providing alink between programming broadcast over digital television broadcastchannels and data that is accessible on demand presents certain problemsand opportunities heretofore unrecognized in the art.

It is within this context that embodiments of the present inventionarise.

SUMMARY OF THE INVENTION

According to an embodiment of the present invention, a method foraccessing ancillary information associated with one or more imagesembedded in a digital broadcast data stream may be implemented in adigital broadcast receiving device configured to receive programming anddata over a plurality of digital broadcast channels. A digitaltelevision broadcast signal may be received by the receiving device. Thedigital broadcast signal may include data representing one or more videoimages and control data associated with one or more visual components ofthe one or more video images. The one or more video images may bepresented on a video display device associated with the digitalbroadcast receiving device. The control data may be used to present aperceptible indication that ancillary information is associated with theone or more visual components in conjunction with presenting the one ormore video images with the display device. The control data may also beused to present the ancillary information with the display device inresponse to an input signal.

In some embodiments, the input signal may be a signal from a remotecontrol associated with the digital broadcast receiving device. In someembodiments, the ancillary information includes one or moreadvertisements.

In some embodiments, the control data may be associated with one or moreobjects depicted in one or more of the video images. The control datamay alternatively be associated with a context associated with the oneor more video images.

In some embodiments, the perceptible indication may be displayed withthe display device while presenting the one or more video images.

In some embodiments, displaying the perceptible indication may includepresenting information guiding a percipient to the ancillary informationthat is associated with the control data using the display device. Insome embodiments, displaying the perceptible indication may includesuperposing the information guiding the percipient to the ancillaryinformation over one or more of the video images. In some embodiments,the information guiding the percipient to the ancillary informationincludes one or more links to one or more resources accessible via anelectronic communications network. In some embodiments, the resourcesmay be accessed via the one or more links in response to the inputsignal.

In some embodiments, the information guiding the percipient to theancillary information may include a video image identifying the one ormore visual components. The image may include one or more links. Eachlink may be associated with one or more of the visual components. Eachlink may be configured to provide access to the ancillary informationassociated with the corresponding visual component.

In some embodiments, displaying the perceptible indication may includepausing presentation of a program containing the one or more videoimages at a particular video image in response to a pause signal. One ormore visual components in that image may be highlighted in a mannerperceptible to a percipient. One or more links to the ancillaryinformation may be associated with the one or more visual components. Insuch embodiments, the ancillary information associated with a particularvisual component may be presented in response to a selection input.

In some embodiments, presenting the ancillary information may includestoring the ancillary information in a memory associated with thedigital broadcast receiving device and/or display device. In otherembodiments, presenting the ancillary information may include using thecontrol data as an input to a search engine coupled to the digitalbroadcast receiving device and/or display device and conducting a searchwith the search engine.

According to another embodiment of the invention, a digital broadcastreceiving device may include a receiver unit, a processor coupled to thereceiver unit and a memory coupled to the processor. The receiver unitmay be configured to receive programming and data over one or moredigital broadcast channels. A set of processor readable instructions maybe embodied in the memory for execution on the processor. Theinstructions may be configured to implement the method described above.

In some embodiments, the digital broadcast receiving device may furthercomprise a video display device, which may be operably coupled to theprocessor.

In some embodiments, the digital broadcast receiving device may furthercomprise a backchannel coupled to the processor. The backchannel mayinclude a network interface coupled to an electronic communicationsnetwork.

In some embodiments, the digital broadcast receiving device may furthercomprise a remote control configured to interoperate with the processor.In such a case the input signal referred to in the method above may be asignal from the remote control.

According to another embodiment, a digital broadcast data stream havingcontrol data associated with one or more images in a digital broadcastdata stream may be provided according to an inventive method. Accordingto the method control data may be associated with one or more visualcomponents of one or more video images. The control data may beconfigured facilitates access to ancillary data associated with the oneor more visual components by a receiving device and/or display deviceusable with a digital broadcast signal generated from the digitalbroadcast data stream. The control data and the video images may becombined into a digital broadcast data stream.

The control data may be configured as in the method described above. Forexample, according to some embodiments, the control data may beassociated with one or more objects depicted in one or more of the videoimages. According to other embodiments, the control data may beassociated with a context of the one or more video images.

According to some embodiments, the digital broadcast data stream may bebroadcast as a digital broadcast signal.

According to some embodiments, the control data may identify one or morehotspots associated with one or more video images. Each hotspot mayprovide a link to ancillary information associated with the one or morevisual components.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 is a schematic diagram illustrating a broadcast environment inwhich embodiments of the present invention may be employed.

FIG. 2 is a block diagram of a digital broadcast receiving deviceaccording to an embodiment of the present invention.

FIG. 3 is a flow diagram illustrating a method for accessing ancillaryinformation associated with an image embedded in a digital broadcastdata stream according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a digital broadcast data stream thatmay be used in conjunction with embodiments of the present invention.

FIGS. 5A-5D illustrate examples of accessing ancillary informationassociated with an image embedded in a digital broadcast data streamaccording to an embodiment of the present invention.

FIG. 6 is a block diagram illustrating method for providing a digitalbroadcast data stream having control data associated with an image in adigital broadcast data stream according to an embodiment of the presentinvention.

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Although the following detailed description contains many specificdetails for the purposes of illustration, anyone of ordinary skill inthe art will appreciate that many variations and alterations to thefollowing details are within the scope of the invention. Accordingly,examples of embodiments of the invention described below are set forthwithout any loss of generality to, and without imposing limitationsupon, the claimed invention.

INTRODUCTION

A digital television broadcast may include control data that allows auser to access ancillary information about visual components of one ormore images in a program that is received as part of a digitaltelevision broadcast.

FIG. 1 schematically illustrates a broadcast environment in whichembodiments of the present invention may be employed. In such anenvironment broadcasters 102 receive or generate input data streams 101.The input data streams 101 are converted to digital broadcast datastreams 103, which are converted to a digital broadcast signal 105 forreception by digital broadcast receivers 110. It is noted that for thesake of example, the two digital broadcast signals 105, 106 are shown asbeing transmitted from different towers 104. This is not to be construedas a limitation upon any embodiment of the invention. Alternatively, thetwo digital broadcast signals 105, 106 may originate from the samebroadcast tower 104 and may be broadcast simultaneously, e.g., usingfrequency division multiplexing.

By way of example, a digital broadcast signal 105 may be a modulatedradiation signal transmitted from a broadcast tower 104, e.g., in theform of an over-the-air broadcast, such as by radiofrequencyelectromagnetic wave signal. It is noted that the digital broadcastsignal 105 transmitted by a given broadcast tower 104 may includemultiple digital broadcast channels modulated at different carriersignal frequencies. Broadcast towers 104 associated with differentbroadcast units 102 may broadcast over different sets of frequencies.For example Broadcaster 1 may broadcast over a first set of carrierfrequencies f₁ . . . f_(m) and Broadcaster N may broadcast over adifferent set of carrier frequencies f′₁ . . . f_(j). There may be someoverlap between the two sets of carrier frequencies.

Furthermore, the digital broadcast signal 105 may be transmitted in aform other than an over-the-air broadcast. Alternatively, embodiments ofthe invention may be used in conjunction with digital broadcaststransmitted over media such as cable (e.g., coaxial cable), opticalfiber, or satellite transmission.

By way of example, the digital broadcast signal 105 may be configured inaccordance with a digital broadcast standard. Examples of digitalbroadcast standards include, but are not limited to, the Digital VideoBroadcasting (DVB) family of standards maintained in Europe andAustralia, the Advanced Television Standards Committee (ATSC) family ofstandards developed for use in the United States and Canada, theIntegrated Services Digital Broadcasting (ISDB) family of standardsdeveloped for use in Japan, Digital Multimedia Broadcasting (DMB)standard used in South Korea.

The DVB family of standards includes the DVB-S and DVB-S2 standards forsatellite television, the DVB-T and DVB-T2 standards for terrestrialtelevision, DVC-C for cable television and DVB-H for mobile televisionand other DVB standards which have been or may be developed. The ATSCfamily of standards includes the ATSC standard for terrestrialtelevision broadcasts and the ATSC M/H standard for broadcasts to mobileand handheld devices. The ISDB family of standards includes the IDSB-S,ISDB-T, and ISDB-C standards, which were developed for satellite,terrestrial and cable television respectively.

By way of example, and not by way of limitation, the digital broadcastsignal 105 may be configured according to the ATSC or ATSC-M/Hstandards. The ATSC standard is described in detail, e.g., in “ATSCDigital Television Standard Part 1—Digital Television System” (A/53,Part 1:2007), “ATSC Digital Television Standard Part 2—RF/TransmissionSystem Characteristics” (A/53, Part 2:2007), and “ATSC DigitalTelevision Standard Part 3—Service Multiplex and Transport SubsystemCharacteristics” (A/53, Part 3, 2007), the disclosures of all three ofwhich are incorporated herein by reference. The ATSC Data BroadcastStandard is described, e.g., in (ATSC Recommended Practice:Implementation Guidelines for the ATSC Data Broadcast Standard (Doc.A/90)”, which is incorporated herein by reference.

The input data stream 101 may include data streams from multiplesources. For example, within the broadcaster 102 data streams fordifferent television program 107 may be made up of audio, video andancillary data streams. These data streams may be multiplexed to form aprogram data stream associated with a given program 107. Multipleprogram data streams may be multiplexed with each other into thebroadcast data stream 103. The digital broadcast data stream 103 mayinclude data representing one or more video images I and control data CDassociated with one or more visual components of the one or more videoimages I.

As used herein, the term “visual component” refers to a visible portionof one or more video images that is associated with an object orcontext. Examples of objects include persons or things depicted within avideo image. Examples of contexts include concepts, persons or thingshaving an association with a person or thing depicted in a video image.For example, a manufacturer of a product may be associated with aproduct or logo depicted in a video image even though the manufactureris not depicted.

Furthermore, one or more data service data streams 108 for broadcastdata services s₁ . . . s_(k) not specifically associated with a givenprogram 107 may be overlaid (e.g., multiplexed) with the program datastreams P₁ . . . P_(N) into the broadcast data stream 103.

The data streams 101 that make up digital broadcast data stream 103 maybe subject to data transforms, such as source coding and compression. Asused herein, “source coding and compression” refers to bit ratereduction methods, also known as data compression, appropriate forapplication to the video, audio, and ancillary digital data streams. Theterm “ancillary data” includes control data, conditional access controldata, and data associated with the program audio and video services,such as closed captioning. “Ancillary data” can also refer toindependent program services. The broadcast unit 102 may include a coderconfigured to minimize the number of bits needed to represent the audioand video information. If configured according to the ATSC standard, thebroadcast unit 102 may employ the MPEG-2 video stream syntax for thecoding of video and the Digital Audio Compression (AC-3) Standard forthe coding of audio.

The broadcast unit 102 may also subject the digital broadcast datastream 103 to service and multiplex transport operations. As usedherein, “service multiplex and transport” refers to the means ofdividing the digital data stream into “packets” of information, themeans of uniquely identifying each packet or packet type, and theappropriate methods of multiplexing video data stream packets, audiodata stream packets, and ancillary data stream packets into a singledata stream. By way of example, and not by way of limitation, digitalbroadcast unit 102 may employ the MPEG-2 transport stream syntax for thepacketization and multiplexing of video, audio, and data signals fordigital broadcasting systems. Such packetization and multiplexing isdescribed e.g., ISO/IEC 13818-1:2000 (E), International Standard,Information technology—Generic coding of moving pictures and associatedaudio information: systems”, which is incorporated herein by reference.

The digital broadcast data stream 103 may be converted to a digitalbroadcast signal 105 through processes referred to as channel coding andmodulation. The channel coder takes the data bit stream encoded in thedigital broadcast data stream 103 and adds additional information thatcan be used by a receiving device 110 to reconstruct the data from thereceived signal which, due to transmission impairments, may notaccurately represent the transmitted signal. A modulation subsystem (orphysical layer) uses the digital data stream information to modulate thetransmitted signal. By way of example and not by way of limitation, theATSC standard, the modulation subsystem offers two modes. Both modes arebased on vestigial sideband modulation. One mode is a terrestrialbroadcast mode known as 8-VSB. The other mode is a high data rate modeknown as 16-VSB.

A digital broadcast receiving device 110 receives the digital broadcastsignal 105 and extracts the digital broadcast data stream 103 includingthe images I and control data CD encoded within the digital broadcastsignal 105. By way of example, and not by way of limitation, the digitalbroadcast receiving device 110 may include a digital receiver 111, aprocessor 114, a memory 115, a display 117 and a data storage device118. The digital broadcast receiving device 110 may be any type ofdevice capable of receiving and utilizing the digital broadcast signal105. By way of example, the digital broadcast receiving device 110 maybe a digital television set, digital radio receiver, personal computer,laptop computer, a mobile or handheld device such as a cellulartelephone, mobile internet device or mobile digital television receiver.Furthermore, the term digital broadcast receiving device encompasses“digital media receivers”, GPS devices, game consoles, portable gamedevices, home, mobile or device security systems, and any combinationthereof and including other devices for which the present invention maybe coupled to provide command and control.

The digital receiver 111 may include one or more tuners 112 and adecoder 113. The tuner(s) 112 may be coupled to an antenna 119 thatreceives the digital broadcast signal 105. The tuner 112 selects one ormore particular frequencies from among the various signals that arepicked up by the antenna 119. The tuner 112 and decoder 113 may extractdata and generate audio and video signals from the digital broadcastsignal 105. By way of example the tuner 112 and decoder 113 may providethe following functions: demodulation, transport stream demultiplexing,decompression, error correction, analog-to-digital conversion, AVsynchronization and media reformatting to fit the specific type ofdisplay 117 optimally.

As used herein, demodulation refers to the process of transforming thereceived digital broadcast signal 105 into a usable signal from whichdata may be extracted and/or from which quality images and sound may beproduced.

Transport Stream Demultiplexing may be implemented, e.g., where multipledigital signals are combined and then transmitted from one antennasource to create over the air broadcasts. In such a case, the decoder113 may decode the digital broadcast data stream 103 and convert it to asuitable form for display of a particular program of the audio and/orvideo components with the display 117 or for extraction of a broadcastservice data stream 108 from the digital broadcast data stream 103. Thedecoder 113 may implement decompression if the digital broadcast datastream 103 contains data in compressed form. The tuner 112 maydecompress such data, e.g., by unpacking compressed packets of digitaldata to their original size.

The decoder 113 may also implement Error Correction to make sure thatany data that is missing from the received digital broadcast signal 105can be corrected. For instance, sometimes interference or a poor-qualitysignal will cause the loss of data information that the tuner 112receives. In such cases, the decoder 112 may perform a number of checksand repair data so that a signal can be viewed on a TV set or data maybe utilized by the processor 114.

The decoder 113 may implement AV Synchronization to coordinate audio andvideo signals being displayed on the display 117 in proper time. AVsynchronization ensures that the audio does not lag behind the videothat is being displayed on the display 117 or vice versa, so that bothaudio and video are in sync. Media reformatting allows the display 117to properly display video images using the data extracted from thedigital broadcast signal. Media reformatting is important since theformatting of images on TV sets may differ significantly according tothe technology employed. For example, some televisions utilize aninterlaced picture, whereas others utilize a progressive-scan picture.

The video images I may be displayed on the video display 117. By way ofexample, the display 117 may be any suitable video and/or audio-visualdisplay usable with the digital broadcast signal 105. By way of example,and not by way of limitation, the display 117 may include a videomonitor, such as a cathode ray tube (CRT), plasma display, liquidcrystal display (LCD) or organic light-emitting diode (OLED) display. Inaddition, the display 117 may include one or more devices for generatingaudio, e.g., one or more speakers.

The processor 114 may be programmed with instructions SW that use thecontrol data CD to present a perceptible indication that ancillaryinformation is associated with one or more visual components of theimages I in conjunction with presenting the images on the display device117. The instructions SW may use the control data CD to present theancillary information with the display device in response to an inputsignal.

As used herein, the term “ancillary information” includes informationabout a visual component of a video image other than the visualdepiction of the component within the image. Such ancillary informationmay include, but is not limited to information identifying visualcomponent, information providing a history of a real object or conceptrepresented by the visual component, advertisements of products orservices associated with the object or concept represented by the visualcomponent.

The receiving device 110 may also implement a backchannel 120 thatallows information to be sent from the device to a broadcast unit 102 ora data center affiliated with the broadcast unit. The back channel 120may be implemented through a digital broadcast signal transmitted fromthe device 110, e.g., via the antenna 119. Alternatively, thebackchannel 120 may be implemented through some other mode ofcommunication such as wireless telephony (e.g., cellular), wirelessinternet, cable, optical fiber and the like. The back channel 120 may beused to retrieve selected ancillary information 122 from a remote source124.

The receiving device 110 may also include a data storage device 118 fornon-volatile storage of data, such as selected ancillary information122. Examples of data storage devices include hard disk drives, flashmemories, compact disk (CD) drives, digital video disk (DVD) drives,tape drives, and the like.

The processor 114 may execute software program instructions thatfacilitate the above-referenced functions. In addition, the processor114 may execute program instructions SW1 . . . SW_(P) for variousdigital broadcast data services. Data for these data services may beupdated as part of the digital broadcast data stream 103 that is carriedby the digital broadcast signal 105. The program instructions SW mayoperate on different portions of control data CD₁ . . . CD_(p) fordifferent groups of related video images within the digital broadcastdata stream 103. These different portions of control data CD₁ . . .CD_(p) may be extracted from the digital broadcast signal 105 andstored, e.g., in the memory 115 or in the data storage 118.

The selection of ancillary information 122 may be filtered from ageographic perspective based on device position information. Tofacilitate such functionality, the receiving device 110 may optionallyinclude a position location system 116, such as a GPS receiver. Forexample, in the case of a mobile or hand-held device, GPS data may beused to filter weather and traffic updates and limit downloads to thosethat are relevant to the area in which the device is presently located.In some embodiments, the function of the position location system 116may be implemented by one of the tuners 112 in conjunction with softwarerunning on the processor 114. The position location signal may originatefrom one or more of the towers 104.

By way of example, a digital broadcast receiving device 200 may beconfigured as shown in FIG. 2 . By way of example, and without loss ofgenerality, the receiving device 200 may be implemented as part of adigital television set, personal computer, video game console, personaldigital assistant, mobile or handheld device such as a cellular phone orpersonal digital assistant, portable email device and the like, or otherdigital device. The device 200 may include a central processing unit(CPU) 205 and a memory 206 coupled to the CPU 205. The CPU 205 may beconfigured to run software applications and, optionally, an operatingsystem. Some embodiments of the present invention may take advantage ofcertain types of processor architecture in which the CPU 205 includes amain processor 205A and one or more auxiliary processors 205B. Eachauxiliary processor may have its own associated local data storage. Oneexample, among others of such a processor architecture is a CellProcessor. An example of a Cell Processor architecture is described indetail, e.g., in Cell Broadband Engine Architecture, copyrightInternational Business Machines Corporation, Sony Computer EntertainmentIncorporated, Toshiba Corporation Aug. 8, 2005 a copy of which may bedownloaded at http://cell.scei.co.jp/, the entire contents of which areincorporated herein by reference.

The memory 206 may store applications and data for use by the CPU 205.The memory 206 may be in the form of an integrated circuit, e.g., RAM,DRAM, ROM, and the like). A computer program 201 may be stored in thememory 206 in the form of instructions that can be executed on theprocessor 205. By way of example, the program 201 may includeinstructions that when executed by the processor cause implement amethod for accessing ancillary information embedded in a digitalbroadcast data stream. The memory 206 may also store Control data 202extracted from a received digital broadcast signal. The control data 202may be used to selectively access ancillary information 203, which mayalso be stored in the memory 206. The memory may also store instructionsfor implementing a search engine 204, e.g., by executing the searchengine instructions with the CPU 205. The search engine 204 may beconfigured, e.g., by suitable programming, to search for particularancillary information 203 using selected control data 202.

The receiving device 200 may also include well-known support functions210, such as input/output (I/O) elements 211, power supplies (P/S) 212,a clock (CLK) 213 and cache 114. The device 200 may further include afast data storage device 215 such as a hard disk drive that providesnon-volatile storage for applications and data. The fast storage device215 may be used for temporary or long-term storage of files 216retrieved from a slower data storage device. By way of example, thestorage device 215 may be a fixed disk drive, removable disk drive,flash memory device, tape drive. Alternatively, storage device 215 maybe, e.g., a CD-ROM, DVD-ROM, Blu-ray, HD-DVD, UMD, or other opticalstorage devices. Files 216 from a slower storage device may betemporarily stored in a faster storage device in a hardware cache forquick loading into the memory 206.

The device 200 may include a digital broadcast receiver 240 which may becoupled to an antenna 241. One or more digital broadcast tuners 242 mayreceive digital broadcast signals picked up by the antenna 241. Thereceiver 240 may further include a decoder 244, which may implement thefunctions described above.

One or more user input devices 220 may be used to communicate userinputs from one or more users to the system 200. By way of example, oneor more of the user input devices 220 may be coupled to the clientdevice 200 via the I/O elements 211. Examples of suitable input device220 include keyboards, mice, joysticks, touch pads, touch screens,remote control units, light pens, still or video cameras, and/ormicrophones.

The client device 200 may include a network interface 225 to facilitatecommunication via an electronic communications network 227. The networkinterface 225 may be configured to implement wired or wirelesscommunication over local area networks and wide area networks such asthe Internet. The system 200 may send and receive data and/or requestsfor files via one or more message packets 226 over the network 227.

The system 200 may further comprise a graphics subsystem 230, which mayinclude a graphics processing unit (GPU) 235 and graphics memory 240.The graphics memory 240 may include a display memory (e.g., a framebuffer) used for storing pixel data for each pixel of an output image.The graphics memory 240 may be integrated in the same device as the GPU235, connected as a separate device with GPU 235, and/or implementedwithin the memory 206. Pixel data may be provided to the graphics memory240 directly from the CPU 205. Alternatively, the graphics unit mayreceive video signal data extracted from a digital broadcast signal fromthe decoder 240. Alternatively, the CPU 205 may provide the GPU 235 withdata and/or instructions defining the desired output images, from whichthe GPU 235 may generate the pixel data of one or more output images.The data and/or instructions defining the desired output images may bestored in memory 206 and/or graphics memory 240. In an embodiment, theGPU 235 may be configured (e.g., by suitable programming or hardwareconfiguration) with 3D rendering capabilities for generating pixel datafor output images from instructions and data defining the geometry,lighting, shading, texturing, motion, and/or camera parameters for ascene. The GPU 235 may further include one or more programmableexecution units capable of executing shader programs.

The graphics subsystem 230 may periodically output pixel data for animage from the graphics memory 240 to be displayed on a video displaydevice 250. The video display device 250 may be any device capable ofdisplaying visual information in response to a signal from the device200, including CRT, LCD, plasma, and OLED displays that can displaytext, numerals, graphical symbols or images. The digital broadcastreceiving device 200 may provide the display device 250 with a displaydriving signal in analog or digital form, depending on the type ofdisplay device. In addition, the display 250 may include one or moreaudio speakers that produce audible or otherwise detectable sounds. Tofacilitate generation of such sounds, the client device 200 may furtherinclude an audio processor 255 adapted to generate analog or digitalaudio output from instructions and/or data provided by the CPU 205,memory 206, and/or storage 215.

The receiving device 200 may optionally include a position locationdevice 270. Such a device may be based on any suitable technologycapable of providing information on the geographic location of a device.Examples of existing technology include global positioning satellite(GPS) technology, inertial guidance technology, and the like.Information from such devices may be used in digital broadcast dataapplications such as navigation for mobile or hand-held devices.

The components of the device 200, including the CPU 205, memory 206,support functions 210, data storage devices 215, user input devices 220,network interface 225, graphics unit 230, audio processor 255 andposition location device 270 may be operably connected to each other viaone or more data buses 260. These components may be implemented inhardware, software or firmware or some combination of two or more ofthese.

By way of example, and not by way of limitation, the program 201 maylocate and present ancillary information in accordance with a method 300depicted in the flow diagram of FIG. 3 . As indicated at 302 thereceiving device 200 may receive a digital television broadcast signal301. The digital broadcast signal 301 includes coded data representingone or more video images 303 and control data 305 associated with one ormore visual components of the images 303. The decoder 244 may convertthe coded control image data 303 into image data usable by the graphicssubsystem 230 and display 250. The decoder 244 may similarly convert thecoded control data 305 to control data 202 usable by the program 201 andCPU 205.

By way of example, the broadcast signal may be configured as depicted inFIG. 4 . Specifically, a digital broadcast signal 400 may include datarepresenting one or more video frame groups 402, 404, 406. Each framegroup may have a group header 408 and a plurality of video frames F. Theframes F₁ . . . F₉ within the frame groups 402, 404, 406 may be relatedto each other within some context. For example, the frames within agiven frame group may be part of particular, scene, shot or sequence ofscenes or shots. The scenes or shots may be related by virtue ofcontaining the same characters or objects or may be related to eachother in terms of where they occur in the course of the plot of aparticular program or episode.

In some embodiments, the control data associated with a correspondingvideo frame group may be included in the group header for that framegroup. By way of example a group header 408 may be associated with aframe group 402 containing data for video frames F₁, F₂ and F₃. By wayof example the group header 408 may be a data packet containing thecontrol data in some coded form that may be utilized by the program 201.Examples of suitable control data include, but are not limited to,information identifying the particular broadcast channel, sub-channel,program or frame group with which the control data is associated.Furthermore, the control data may identify particular visual componentswithin the frames F₁, F₂, F₃ that make up the frame group 402. Inaddition, the control data may identify a location of ancillaryinformation associated with one or more visual components of the framesF.

By way of example, program 201 may include a network protocol stack thatmay be used to facilitate communication over the network 227. Thecontrol data in the header 408 may include mark-up language code (e.g.,HTML, XML and the like) that provides a universal resource locator (URL)for network resource (e.g., a website) from which ancillary informationmay be obtained. Alternatively, the control data in the header 408 mayinclude text descriptors that may be used as inputs to the search engine204.

As indicated at 304, the video images corresponding to the image data303 may be presented, e.g., using the display device 250. As indicatedat 306, the program 201 may use the control data 202 to present aperceptible indication that ancillary information is associated with theone or more visual components in conjunction with presenting the one ormore video images with the display device. For example, as shown in FIG.5A, the display 250 may display a symbol 502 that is associated with thepresence of ancillary information associated with a video image 504 (orgroup of such images) that is concurrently displayed. The presence of asymbol notifies a percipient 506 of the availability of the ancillaryinformation. For the purposes of this disclosure, a percipient is onewhose senses are stimulated by an output from the display device 250.

In some embodiments, the program 201 may optionally be configured topresent guiding information that directs the percipient 506 to theancillary information, as indicated at 308. As indicated at 310, theprogram 201 may use the control data 202 to present or store theancillary information in response to an input signal 309. By way ofexample, and not by way of limitation, the control data 305 associatedwith the image 504 may include a URL for a web page associated with aframe group that includes the image 504. The web page may display theancillary information in any suitable form. For example, the ancillaryinformation may be in the form of links to various sources ofinformation related to the visual components of the image 504. Suchlinks may be grouped according to their association with a particularvisual component of the image 504. Such links may be activated, e.g.,through a signal from the interface 220.

For example, the user interface 220 may be in the form of a remotecontrol of a type commonly used with a television set or DVR. Theinterface 220 may include a pause button 221 and one or more controlbuttons 223 which may include arrow buttons and numbered buttons. Thepercipient 506 may see something of interest in the image 504. In such acase, the percipient 506 may activate the pause button 221 to freeze thedisplay of the program of which the image 504 is a part. Ancillaryinformation 203 may be presented by the display 250 or stored in thememory 206 or storage 215 in response to a signal initiated by thepercipient's activation of the pause button 221 or by a signal generatedin response to activation of one or more of the control buttons 223 orsome combination of both.

The program 201 may use the control data 202 to access ancillaryinformation 203 residing in the memory 206 or storage 215 or remoteancillary information residing on one or more servers 229 coupled to thenetwork 227. The program 201 may utilize one or more portions of thecontrol data 202 as input to a search engine 204 resident in memory 206or may access a remote search engine server 228 via the networkinterface 226 and the network 227 for this purpose. By way of example,the control data 202 may include text descriptors associated with visualcomponents of the video images as inputs to the search engine 204 or thesearch engine server 229.

There are a number of ways in which guiding information or ancillaryinformation may be presented. For example, suppose that the image 504depicts an automobile 510, a soft drink can 512, a laptop computer 514,an actor 516 portraying a character and wearing clothes 518 and shoes520. Upon receipt of the signal from the interface 220, the program 201may use the control data 202 associated with the image 504 to cause thedisplay 250 to present a menu 530, as shown in FIG. 5B containinglistings 510B, 512B, 514B and 516B for the automobile 510, soft drinkcan 512, laptop 514 and actor 516. Each of these listings may provideancillary information (e.g., in the form of text or images) or guidinginformation associated with ancillary information. Guiding informationmay be in the form of links to information resident in the memory 206,storage 215 or at the remote server 229. In the example depicted in FIG.5B a particular listing may be selected, e.g., using left-right orup-down arrow keys on the interface 220. A given link within a listingmay be activated by pressing a numbered control button on the interface220 corresponding to a number listed next to the link.

By way of example, listing 510B for the automobile 510 may include linksto web pages describing the history of the particular make and model ofautomobile, an advertisement for a dealership that sells that particularmake or model, or for a dealership that sells the tire depicted on theautomobile 510. The listing 512B for the soft drink can 512 may includelinks to the web page of the soft drink company, an advertisement forthe soft drink or a coupon for the soft drink. The listing 514B for thelaptop computer 514 may include a links to websites for the laptopmanufacturer or dealer, reviews of the particular brand of laptopdepicted or technical help for the laptop. The listing 516B for theactor 516 may include links to information regarding either the actor orthe character portrayed or information regarding the clothes 518 orshoes 520 that the actor is wearing in the image 504.

In some embodiments, information guiding the percipient to the ancillaryinformation may be superposed over one or more of the video imagespresented by the display. For example, as shown in FIG. 5C, a programmay be paused at a particular frame 505, in response to a signal fromthe input device 220 that is generated when the pause button 221 isactivated. The program 201 may highlight certain visual componentswithin the image in a manner perceptible to a percipient 506 if thosecomponents have ancillary information associated with them. Highlightedregions of a video image that are associated with ancillary data aresometimes referred to herein as “hotspots”. By way of example, and notby way of limitation, as shown in FIG. 5C, portions of the image 504containing the automobile 510, soft drink can 512, laptop 514, and actor516 may be surrounded by dashed polygons 510′, 512′, 514′ and 516′, thatmark the boundaries of hotspots. In addition, the actor's clothes 518and shoes 520 may be shaded to stand out from the rest of the image 504,thereby identifying these regions as hotspots.

Digital links to the ancillary information associated with these visualcomponents may be associated with the corresponding portions of theimage 504 that mark the hotspots. The ancillary information associatedwith a particular visual component may be presented in response to aselection input. For example, the input device may include selectionbuttons that can be used to select the particular image portion.Alternatively, the display device 250 may include a touch screen onwhich the image 504 is displayed. In such a case, a particularhighlighted portion of the image 504 may be selected by touching thecorresponding region of the touch screen on which it is presented. Theinformation guiding the percipient to the ancillary information (or theancillary information itself) may be presented in the form of “bubbletext” 522 that appears next to the corresponding portion of the image504. The “bubble text” 522 may include one or more links 524 toresources containing the ancillary information.

In some embodiments, ancillary information (or guiding information) maybe stored for later review. For example, ancillary information retrievedfrom the remote source 229 may be stored in the memory 206 or thestorage device 215. In some implementations of this embodiment, storedancillary information (or guiding information) may be subsequentlyretrieved. By way of example, and not by way of limitation, FIG. 5Dillustrates an example of a presentation of saved guiding information540 presented on the display device 250. In this example, the guidinginformation is in the form of a plurality of menu lists 542. Each menulist may correspond to a particular program or episode of a particularprogram. The lists may be broken down by frame group, with each listcorresponding to a different frame group. The program 201 may generatethe lists as the percipient 506 selects ancillary information or guidinginformation to be saved. The lists 542 may include links to theancillary information (or guiding information) for the frame group. Forexample, selecting the link marked group 1A may cause the program 201 topresent a menu, such as the menu 530 depicted in FIG. 5B.

According to another embodiment of the present invention a digitalbroadcast data stream may be provided as illustrated with respect toFIG. 6 . According to a method 600, video control data 602 may beassociated with one or more images 604 in a digital broadcast datastream 608. By way of example, the control data may be associated withone or more visual components of one or more video images using aneditor 606. By way of example, the editor may introduce time-varyingsensitive regions into portions of one or more video images. Examples oftechniques for introducing such sensitive regions are described, e.g.,in “Time-Varying Sensitive Regions in Dynamic Multimedia Objects: aPragmatic Approach to Content-Based Retrieval from Video” by VictoriaBurrill, Thomas Kirste and Jochen Weiss, Information and SoftwareTechnology 1994, Volume 36, Number 4, the entire contents of which areincorporated herein by reference.

The digital broadcast data stream 608 may be coupled to a digitalbroadcasting unit 610, which converts the digital broadcast data stream608 to a digital broadcast signal 612. The broadcasting unit 610 may beany system commonly used to convert data to a format suitable forbroadcast according to a digital broadcasting standard, such as the ATSCstandard. The broadcasting unit may include a transmitter and broadcasttower with an antenna for transmitting the signal 612 over the airwaves.Alternatively, the broadcast signal 612 may be transmitted to a separatetransmitting unit for this purpose.

The control data 602 may be configured to facilitate access to ancillarydata associated with the visual components when the digital broadcastsignal 612 is received and/or the video images 604 are displayed, asdiscussed above. By way of example, the control data 602 may beassociated with one or more objects depicted in one or more of the videoimages or a context of the one or more video images. The control data602 may identify one or more hotspots associated with one or more of thevideo images 604. Each hotspot may provide a link to ancillaryinformation associated with one or more visual components in thehotspot. The control data 602 may be configured to cause a displaydevice to present guiding information, as discussed above.

While the above is a complete description of the preferred embodiment ofthe present invention, it is possible to use various alternatives,modifications and equivalents. Therefore, the scope of the presentinvention should be determined not with reference to the abovedescription but should, instead, be determined with reference to theappended claims, along with their full scope of equivalents. Any featuredescribed herein, whether preferred or not, may be combined with anyother feature described herein, whether preferred or not. In the claimsthat follow, the indefinite article “A”, or “An” refers to a quantity ofone or more of the item following the article, except where expresslystated otherwise. The appended claims are not to be interpreted asincluding means-plus-function limitations, unless such a limitation isexplicitly recited in a given claim using the phrase “means for”.

The invention claimed is:
 1. A method for providing a digital broadcastdata stream having control data associated with one or more video imagesin the digital broadcast data stream, the method comprising: a)associating control data with one or more visual components of one ormore video images, wherein the control data facilitates access toancillary data associated with the one or more visual components by adisplay device, the control data provided in a group header identifiesancillary data location information for retrieving the ancillary datarelevant to geographic location of the display device for presenting onthe display device; and b) combining the control data and the videoimages into a combined digital broadcast data stream for transmitting tothe display device for rendering, the combining enables decoding at thedisplay device such that the ancillary data is retrieved and overlaidonto the one or more video images when rendered on the display device,wherein the control data limits downloads by facilitating access toselect ones of ancillary data that are relevant to the geographiclocation of the display device by dynamically filtering the ancillarydata at an ancillary data location accessed using the ancillary datalocation information included in the group header and embedding links tothe select ones of the ancillary data relevant to the geographiclocation of the display device in the control data included in thecombined digital broadcast data stream transmitted to the displaydevice, wherein the links activate the access to download the selectones of the ancillary data to the display device by dynamicallyimplementing a back channel communication from the display device to theancillary data location, when the one or more video images are to berendered, the select ones of the ancillary data provide access toadditional information for the one or more visual components included inthe one or more video images of the digital broadcast data stream, theadditional information being relevant to the geographic location of thedisplay device.
 2. The method of claim 1, wherein the control data isassociated with one or more objects depicted in one or more of the videoimages; or wherein the control data is associated with a context of theone or more video images; or wherein the control data identifies one ormore hotspots associated with one or more video images, wherein eachhotspot provides a link to ancillary data associated with the one ormore visual components; or wherein the control data is configured tocause the display device to present information guiding a percipient tothe ancillary data associated with the control data; or wherein thecontrol data is configured to pause presentation of a program containingthe one or more video images at a particular image of the one or morevideo images in response to a pause signal, highlight the one or morevisual components in a manner perceptible to the percipient, andassociate one or more links to the ancillary data associated with theone or more visual components; or wherein the control data is configuredto cause the display device to present a perceptible indication thatancillary data is associated with the one or more visual components inconjunction with presenting the one or more video images with thedisplay device; or wherein the control data includes input for a searchengine.
 3. The method of claim 1, wherein the ancillary data includesone or more advertisements.
 4. The method of claim 1, furthercomprising, c) broadcasting the combined digital broadcast data streamas a digital broadcast signal.
 5. The method of claim 1, wherein thelinks embedded in the control data is configured to cause the displaydevice to present information guiding a percipient to the ancillary dataassociated with the one or more links in the control data, wherein thecontrol data is configured to superpose the information guiding thepercipient to the ancillary data over one or more of the video images;wherein the information guiding the percipient to the ancillary dataincludes a video image identifying the one or more visual components,wherein the video image is associated with the one or more links, eachlink being associated with one or more of the visual components, whereineach link is configured to provide access to the ancillary dataassociated with the corresponding visual component.
 6. The method ofclaim 1, wherein the control data is configured to pause presentation ofa program containing the one or more video images at a particular videoimage of the one or more video images in response to a pause signal,highlight the one or more visual components in a manner perceptible to apercipient, and associate one or more links to the ancillary dataassociated with the one or more visual components, and wherein thecontrol data is configured to present the ancillary data associated witha particular visual component of the one or more visual components inresponse to a selection input.
 7. The method of claim 1, wherein thecontrol data is configured to cause the display to present a perceptibleindication that ancillary data is associated with the one or more visualcomponents in conjunction with presenting the one or more video imageswith the display device, wherein the control data is configured to causethe display device to present the perceptible indication whilepresenting the one or more video images.
 8. The method of claim 1,wherein the control data is configured to cause the display device topresent information guiding a percipient to the ancillary dataassociated with the control data, wherein the control data is configuredto cause the display device to superpose the information guiding thepercipient to the ancillary data over one or more of the video images.9. The method of claim 1, wherein the control data is configured tocause the display device to present information guiding a percipient tothe ancillary data associated with the control data, wherein theinformation guiding the percipient to the ancillary data includes one ormore links to one or more resources accessible via an electroniccommunications network.
 10. The method of claim 9, wherein the one ormore links are configured to be activated in response to the inputsignal.
 11. The method of claim 1, wherein the control data isconfigured to cause the display device to present information guiding apercipient to the ancillary data associated with the control data,wherein the information guiding the percipient to the ancillary dataincludes a video image identifying the one or more visual components,wherein the video image includes one or more links, each link beingassociated with one or more of the visual components, wherein each linkis configured to provide access to the ancillary data associated withthe corresponding visual component.
 12. A digital broadcast receivingdevice, comprising: a receiver configured to receive programming anddata over one or more digital broadcast channels; a processor coupled tothe receiver; a memory coupled to the processor; and a set of computerexecutable instructions embodied in the memory for execution on theprocessor, the instructions configured to implement a method foraccessing control data associated with one or more video images embeddedin a digital broadcast data stream, the method comprising: a)associating the control data with one or more visual components of oneor more video images, wherein the control data facilitates access toancillary data associated with the one or more visual components by adisplay device, the control data provided in a group header thatidentifies ancillary data location information for retrieving theancillary data relevant to geographic location of the display device forpresenting on the display device; b) combining the control data and thevideo images into a combined digital broadcast data stream, thecombining enables decoding at the display device such that the ancillarydata relevant to the geographic location is retrieved and overlaid ontothe one or more video images when rendered on the display device,wherein the control data limits downloads by facilitating access toselect ones of ancillary data that are relevant to the geographiclocation of the display device by dynamically filtering the ancillarydata at an ancillary data location accessed using the ancillary datalocation information included in the group header and embedding one ormore links to the select ones of the ancillary data relevant to thegeographic location of the display device in the control data includedin the combined digital broadcast data stream transmitted to the digitalbroadcast receiving device, wherein the links activate the access todownload the select ones of the ancillary data to the display device bydynamically implementing a back channel communication from the displaydevice to the ancillary data location, when the one or more video imagesare to be rendered, the select ones of the ancillary data provide accessto additional information for the one or more visual components includedin the one or more video images of the digital broadcast data stream,the additional information being relevant to the geographic location ofthe display device.